fix: show LLM provenance in automation output by enyst · Pull Request #547 · OpenHands/extensions

enyst · 2026-09-09T22:18:25Z

HUMAN:
This PR proposes a fix to show the LLM in automations. Which LLM is running can be relevant, and it is for us specially with PR review, as they don’t all have the same behavior.

A human has tested these changes.

Why

PR-review and Slack automation results do not identify the LLM profile and concrete model that generated them, making model-dependent judgments difficult to calibrate. The output implementations live in OpenHands/extensions, while the linked issue was filed in the Automation service repository.

Summary

Use the concrete active/default Agent Server settings for spawned reviewer and Slack conversations, and derive the displayed profile/model from that same configuration.
Append the same concise footer everywhere: LLM profile: <profile> · Model: <model>.
Cover GitHub review instructions/fallbacks and Slack completion output with regression tests; bump and regenerate the reviewer bundle to 1.0.1.

Issue Number

Tracks OpenHands/automation#430

Fixes #548

How to Test

uv sync --group test
npm run build:automations
uv run pytest -q
uv run python scripts/sync_extensions.py --check
uv run python -m py_compile skills/github-pr-reviewer/scripts/main.py skills/slack-channel-monitor/scripts/main.py
git diff --check

Expected result: 802 passed, 24 skipped. The sync check emits one pre-existing, non-blocking coverage warning for plugins/issue-duplicate-checker.

Video/Screenshots

Not applicable; this changes generated GitHub/Slack text output and includes automated request-body assertions.

Notes

The existing /api/settings plaintext request is required to serialize the child Agent's LLM credential back to the same authenticated Agent Server; this PR no longer adds named-profile plaintext reads.
The linked issue was classified as enhancement and marked ready-for-dev after confirming it already contained the required Desired Behavior and Acceptance Criteria sections.
Successful GitHub reviews are posted by the review agent, so the exact footer is required in its deterministic publishing prompt. Script-owned fallback and error comments append it directly.

This pull request was created by an AI agent (OpenHands) on behalf of the user.

@enyst can click here to continue refining the PR

Co-authored-by: openhands <openhands@all-hands.dev>

all-hands-bot · 2026-09-09T23:49:35Z

🤖 OpenHands is reviewing this PR.

Head commit: 7a0dba57a6454c4163f71b8e3711405238fb6b84
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/b5176e74-a8ae-4f4f-a5b9-70d0bb4c5986

This comment was posted by an AI agent (OpenHands).

all-hands-bot · 2026-09-09T23:49:36Z

🤖 OpenHands is reviewing this PR.

Head commit: 7a0dba57a6454c4163f71b8e3711405238fb6b84
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/023d9f90-1538-4503-8e98-2af81e1ac368

This comment was posted by an AI agent (OpenHands).

all-hands-bot · 2026-09-09T23:52:18Z

⚠️ OpenHands PR Reviewer encountered a problem at commit 7a0dba57a645 (status: error).

This comment was posted by an AI agent (OpenHands).

enyst · 2026-09-10T03:12:42Z

@OpenHands /codereview this pr and post your review as a comment.

openhands-ai · 2026-09-10T03:13:03Z

I'm on it! enyst can track my progress at all-hands.dev

enyst · 2026-09-10T03:31:46Z

This review comment was generated by an AI agent (OpenHands) on behalf of @enyst.

🔴 Needs improvement

[CRITICAL ISSUES]

[skills/github-pr-reviewer/scripts/main.py:650-655, skills/slack-channel-monitor/scripts/main.py:528-533] A renamed or deleted selected profile disables both automations instead of falling back. AUTOMATION_MODEL is a persisted profile name, so it can become stale. Both implementations call _fetch_llm_profile() without handling its 404; the exception then aborts conversation creation. This contradicts the repository's documented custom-automation contract in skills/openhands-automation/references/custom-automation.md:279-295, which requires falling back to the user's default LLM when the selected profile no longer exists. I reproduced this in both modules: a profile endpoint 404 propagates as HTTPError rather than selecting the default. Catch only the not-found case, use the already-fetched default agent_settings.llm, record the actual fallback profile/model, and keep authentication/server/malformed-response failures fatal.
[skills/github-pr-reviewer/scripts/main.py:458-477, skills/github-pr-reviewer/scripts/main.py:1057-1058] The successful GitHub path does not guarantee the feature it claims to provide. The script asks the LLM to append the footer, but _matching_review_exists() accepts any review by the bot at the head commit without inspecting its body. I reproduced a matching review whose body was review without provenance; _matching_review_exists() returned True, so the automation closes the record and emits no deterministic fallback. Prompt compliance is not an invariant. Make publication script-owned so the footer is appended before the GitHub request, or at minimum validate the matched review body and deterministically publish the missing provenance before marking the task closed.

[TESTING GAPS]

[skills/github-pr-reviewer/tests/test_main.py:533-601, tests/test_slack_channel_monitor.py:57-106] The new tests cover successful profile resolution, prompt text, and fallback comments, but not either failure above. Add regression tests for a selected-profile 404 in both modules and for a matching GitHub review that omits or alters the required footer.

All eight current PR checks pass. Local py_compile and git diff --check also pass; the focused reproductions above exercise the uncovered behavior directly.

[RISK ASSESSMENT]

[Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM

This changes LLM selection and external output in two automation paths. There are no dependency changes or new external services, but stale profile state can stop both automations and the primary GitHub success path remains nondeterministic.

VERDICT:
❌ Needs rework: The selected-profile fallback and successful-review provenance guarantee must be fixed before merge.

KEY INSIGHT:
Model provenance is audit metadata, so the deterministic automation layer - not the LLM prompt - must own and verify it.

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.

Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.

When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

openhands-ai · 2026-09-10T03:32:06Z

Reviewed PR #547 and posted the review comment:

#547 (comment)

The review requests changes for two issues:

Deleted or renamed AUTOMATION_MODEL profiles return 404 and stop both automations instead of falling back to the default LLM.
The GitHub success path accepts any matching review without verifying the required provenance footer, so prompt noncompliance can silently omit it.

All 8 current PR checks are green. py_compile and git diff --check passed. Focused reproductions confirmed both findings. Local pytest and sync checks could not run because the existing environment lacks pytest and PyYAML.

enyst · 2026-09-10T04:51:08Z

Astra feedback: request changes, linked provider profiles lose resolved credentials.

Address profile fallback and successful-review footer findings from #547. Fail clearly if linked credentials need the accompanying Agent Server fix. Co-authored-by: openhands <openhands@all-hands.dev>

The active profile pointer can drift from the concrete default LLM settings. Label 404 fallback as default instead of claiming the named profile was read. Co-authored-by: openhands <openhands@all-hands.dev>

github-actions · 2026-09-11T11:58:17Z

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. The directory will be automatically removed when the PR is approved.

all-hands-bot · 2026-09-11T11:58:23Z

🤖 OpenHands is reviewing this PR.

Head commit: f899e743b5c970dd35436c40b048823f40a16a94
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/bfd3bc1b-b8d7-4973-b583-f58e64bae300

This comment was posted by an AI agent (OpenHands).

all-hands-bot · 2026-09-11T12:01:49Z

⚠️ OpenHands PR Reviewer encountered a problem at commit f899e743b5c9 (status: error).

This comment was posted by an AI agent (OpenHands).

Co-authored-by: openhands <openhands@all-hands.dev>

fix: show LLM provenance in automation output

7a0dba5

Co-authored-by: openhands <openhands@all-hands.dev>

github-actions Bot added the type: fix A bug fix label Sep 9, 2026

openhands-ai Bot mentioned this pull request Sep 9, 2026

Always display the LLM profile in use in PR-review and Slack automation output OpenHands/automation#430

Open

4 tasks

enyst marked this pull request as ready for review September 9, 2026 23:49

enyst requested a review from all-hands-bot September 9, 2026 23:49

all-hands-bot removed their request for review September 9, 2026 23:52

enyst requested a review from all-hands-bot September 10, 2026 03:12

fix: make automation provenance and publication reliable

49dd465

Address profile fallback and successful-review footer findings from #547. Fail clearly if linked credentials need the accompanying Agent Server fix. Co-authored-by: openhands <openhands@all-hands.dev>

enyst mentioned this pull request Sep 10, 2026

fix(agent-server): resolve linked providers for plaintext profile reads enyst/agent-sdk#28

Closed

1 task

fix: identify fallback settings without assuming the active profile

f899e74

The active profile pointer can drift from the concrete default LLM settings. Label 404 fallback as default instead of claiming the named profile was read. Co-authored-by: openhands <openhands@all-hands.dev>

enyst mentioned this pull request Sep 10, 2026

fix: address provenance review findings for OpenHands/extensions #547 enyst/oh-extensions#2

Closed

1 task

enyst mentioned this pull request Sep 11, 2026

fix(agent-server): resolve linked providers for plaintext profile reads OpenHands/software-agent-sdk#4952

Draft

1 task

all-hands-bot removed their request for review September 11, 2026 12:01

enyst marked this pull request as draft September 11, 2026 12:29

fix: use active settings for LLM provenance

1b6a4a3

Co-authored-by: openhands <openhands@all-hands.dev>

Navigation Menu

Uh oh!

fix: show LLM provenance in automation output - #547

fix: show LLM provenance in automation output#547
enyst wants to merge 4 commits into
mainfrom
openhands/show-llm-provenance

enyst commented Sep 9, 2026 •

edited

Loading

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

openhands-ai Bot commented Sep 10, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

openhands-ai Bot commented Sep 10, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

all-hands-bot commented Sep 11, 2026

Uh oh!

all-hands-bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

enyst commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Summary

Issue Number

How to Test

Video/Screenshots

Notes

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

all-hands-bot commented Sep 9, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

openhands-ai Bot commented Sep 10, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

openhands-ai Bot commented Sep 10, 2026

Uh oh!

enyst commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

all-hands-bot commented Sep 11, 2026

Uh oh!

all-hands-bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

enyst commented Sep 9, 2026 •

edited

Loading